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EXECUTIVE  SUMMARY 


BACKGROUND 

The  Badlands  Bombing  Range  (BBR)  is  located  in  the  southwest  part  of  South 
Dakota.  It  spans  some  300,000  acres  and  was  primarily  in  use  by  the  Defense  Department 
from  World  War  II  until  the  1960s.  Since  that  time,  parts  of  the  former  bombing  range, 
including  several  impact  areas  heavily  contaminated  with  ordnance  and  numerous 
uncharacterized  areas,  have  been  returned  to  a  Sioux  tribe,  who  has  an  interest  in  locating 
ordnance  and  cleaning  the  sites.  Under  funding  from  the  Native  American  Lands 
program,  the  Environmental  Security  Technology  Certification  Program  (ESTCP)  has 
sponsored  demonstrations  of  advanced  sensor  technologies  in  recent  years  to  characterize 
the  ordnance  contamination  on  portions  of  this  site. 

Two  of  these  demonstrations  will  be  discussed  in  this  report.  The  first,  conducted 
in  1997  by  the  Naval  Research  Lab,  used  the  Multisensor  Towed  Array  Detection  System 
(MTADS).  The  second,  conducted  in  1999  by  the  Army  Corps  of  Engineers  in 
conjunction  with  Oak  Ridge  National  Laboratory,  used  a  high-sense  helicopter-mounted 
magnetic  mapping  (HM3)  system.  This  work  is  an  independent  evaluation  of  the 
perfonnance  of  these  two  advanced  sensor  systems,  both  individually  and  in  comparison 
to  one  another. 

CONCLUSIONS 

The  most  appropriate  use  for  these  technologies  may  be  in  different  missions.  The 
HM3  mounted  on  a  helicopter  was  originally  conceived  as  footprint  reduction  tool,  where 
the  main  requirement  is  to  detect  enough  ordnance  or  ordnance-related  debris  to  identify 
areas  that  warrant  more  thorough  examination.  On  the  other  hand,  the  MTADS’  role  is  to 
produce  dig  lists  of  individual  targets.  As  such,  the  ability  to  do  discrimination  is  not 
equally  important  to  the  two  systems.  In  fact,  for  the  footprint  reduction  mission,  finding 
ordnance-related  clutter  may  provide  as  important  an  indication  of  an  impact  area  as 
finding  intact  ordnance. 
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The  results  of  the  HM3  demonstration  were  very  good,  even  for  detecting 
individual  targets.  Using  MTADS  as  a  comparison  standard  for  detection  and  discrimi¬ 
nation,  we  conclude  the  following: 

•  On  a  homogeneous  site  the  Pd  achieved  by  the  HM3  on  individual  targets  is 
about  50  percent  of  that  achieved  by  the  MTADS. 

•  Only  about  8  percent  of  the  apparent  clutter  that  appears  in  the  MTADS  dig 
lists  is  reported  by  the  HM3. 

•  Of  22  ordnance  items  detected  and  confirmed  in  the  ground  truth  by  HM3,  20 
were  detected  by  MTADS.  The  cause  of  one  missed  detection  is  likely 
inaccessibility  of  the  area  by  the  vehicle. 

•  The  helicopter-mounted  HM3  provides  much  faster  production.  We  roughly 
estimate  that  the  HM3  survey  rate  would  be  about  10  times  faster  that  the 
MTADS  rate  for  a  large  site. 

•  Cost  estimates  prepared  by  the  performers  indicate  that  the  per  acre  cost  of 
the  MTADS  is  about  2-3  times  higher  that  those  of  the  HM3.  These  figures 
are  very  rough  estimates  and  may  not  accurately  reflect  the  cost  differences 
seen  in  operational  surveys. 

RECOMMENDATIONS 

In  the  process  of  this  analysis,  we  have  encountered  several  questions  that  could 
not  be  addressed  adequately  with  that  data  as  it  was  collected.  This  is  not  surprising  since 
the  purpose  of  neither  data  collection  was  to  provide  comparison  data.  Nevertheless,  we 
offer  some  suggestions  for  future  similar  efforts  that  would  allow  for  more  complete 
analysis: 

•  The  ground  truth  from  the  HM3  excavated  targets  was  inadequate,  both  for 
evaluation  of  this  system  and  for  comparison  with  the  MTADS.  Although 
extensive  ground  truth  sampling  is  expensive,  it  is  critical  to  obtaining 
meaningful  data  for  live  demonstrations.  Sampling  of  at  least  100  targets  in 
future  demonstrations  is  recommended. 

•  The  elapsed  time  of  2  years  between  the  MTADS  and  HM3  demonstrations 
likely  had  some  impact  on  the  system  comparisons.  This  site  was  in  use 
during  the  intervening  time,  when  items  producing  magnetic  anomalies  may 
have  been  added,  removed,  or  displaced  by  farming  activities. 

•  Conducting  excavations  between  the  two  demonstrations  caused  complica¬ 
tions.  Obviously,  this  expenditure  of  resources  provided  ground  truth  for  only 
one,  instead  of  both,  demonstrations.  It  also  made  data  analysis  for 
comparisons  more  time  consuming,  requiring  the  tedious  separation  of 
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MTADS  targets  to  isolate  those  left  in  the  ground  and  available  to  be 
detected  by  the  HM3. 

Finally,  the  selection  of  targets  for  ground  truthing  should  be  performed  with 
a  well  articulated  purpose  in  mind.  It  is  difficult  to  make  meaningful 
comparisons  if  the  goal  of  one  system  is  to  maximize  ordnance  picks  and  that 
of  the  other  is  to  span  the  range  of  signals  for  analysis  purposes. 
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REVIEW  OF  UNEXPLODED  ORDNANCE  DETECTION 
DEMONSTRATIONS  AT  THE  BADLANDS  BOMBING  RANGE 


NRL  Multisensor  Towed  Array  Detection  System  (MTADS) 

and 

ORNL  High-sense  Helicopter-Mounted  Magnetic  Mapping  (HM3)  System 
A.  INTRODUCTION 

The  Badlands  Bombing  Range  (BBR)  is  located  in  the  southwest  part  of  South 
Dakota.  Spanning  some  300,000  acres,  the  range  was  primarily  in  use  by  the  Defense 
Department  from  World  War  II  until  the  1960s.  Portions  of  the  range  were  used  by  the 
Air  Force  as  bombing  targets,  and  other  portions  were  used  for  ground-fired  weapons. 
Since  that  time,  parts  of  the  former  bombing  range,  including  several  impact  areas 
heavily  contaminated  with  ordnance  and  numerous  uncharacterized  areas,  have  been 
returned  to  a  Sioux  tribe,  who  has  an  interest  in  locating  ordnance  and  cleaning  the  sites. 
Under  funding  from  the  Native  American  Lands  program,  the  Environmental  Security 
Technology  Certification  Program  (ESTCP)  has  sponsored  demonstrations  of  advanced 
sensor  technologies  in  recent  years  to  characterize  the  ordnance  contamination  on 
portions  of  this  site. 

Two  of  these  demonstrations  will  be  discussed  in  this  report.  The  first,  conducted 
in  1997  by  the  Naval  Research  Laboratory  (NRL),  used  the  Multisensor  Towed  Array 
Detection  System  (MTADS).  The  second,  conducted  in  1999  by  the  Army  Corps  of 
Engineers  in  conjunction  with  Oak  Ridge  National  Laboratory  (ORNL),  used  a  high- 
sense  helicopter-mounted  magnetic  mapping  (HM3)  system.  The  results  of  these  demon¬ 
strations  were  reported  separately  by  their  respective  performers  (Refs.  1  and  2.)  Both 
systems  surveyed  two  common  areas,  and  the  purpose  of  this  work  is  to  compare  the 
performance  of  the  two  systems. 

In  1997,  NRL  visited  the  BBR  and  took  survey  data  on  two  sites  in  the  Cuny 
Table  area,  hereafter  referred  to  as  BBR1  and  BBR2,  using  MTADS.  Both  sites  were 
former  impact  areas.  BBR1  was  used  chiefly  as  an  aerial  bombing  target  and  contained 
(almost)  exclusively  M-38  practice  bombs  and  their  associated  scrap.  The  M-38  practice 
bomb,  while  not  carrying  a  live  warhead,  does  carry  a  2-pound  black  powder  spotting 
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charge.  The  BBR1  site  is  divided  by  a  fence  along  an  approximately  east-west  line 
bisecting  the  former  bull’s-eye.  The  section  north  of  the  fence  has  been  used  for  livestock 
grazing  and  was  entirely  accessible  for  both  demonstrations.  The  section  south  of  the 
fence  has  been  cultivated  for  farming.  As  a  result,  access  was  somewhat  impeded  by 
crops  on  the  south  side  of  the  site  during  the  MTADS  survey,  and  excavation  of  targets 
was  not  possible  for  either  demonstration.  The  BBR2  area  contained  more  varied 
ordnance,  including  bombs  and  rockets. 

MTADS  is  a  surface-towed  ordnance-detection  system  that  supports  both  full- 
field  cesium-vapor  magnetometer  and  electromagnetic  induction  (EMI)  arrays.  This  study 
is  concerned  with  the  magnetometer  data  only.  Typical  sensor  height  above  ground  for 
the  magnetometers  is  0.25  m,  typical  line  spacing  is  0.25  m,  and  typical  along-track  data 
density  is  0.05  m  per  point.  High-quality  differential  Global  Positioning  System  (DGPS) 
geo-referenced  digital  data  is  taken  at  high  spatial  density  and  processed  using  a  set  of 
specially  developed  algorithms  to  fit  magnetometer  and/or  EMI  signatures  of  operator- 
selected  anomalies.  An  initial  set  of  targets  spanning  the  range  of  target  sizes  and  types 
was  used  as  a  training  set  to  guide  target  selection.  The  MTADS  analysis  resulted  in  dig 
lists  containing  704  magnetometer  targets  (485  north  of  the  fence  and  219  south)  for 
BBR1  and  647  for  BBR2.  Of  these  dig  list  targets,  146  targets  on  BBR1  and  255  targets 
on  BBR2  were  excavated.  Details  of  the  sensor  and  the  NRL  site  activities  are  provided 
in  Reference  1 . 

In  1999,  ORNL  collected  DGPS  geo-referenced  data  at  the  BBR  Cuny  Table  area 
with  a  helicopter-mounted  HM3  full-field  cesium-vapor  magnetometer  system.  Com¬ 
pared  with  MTADS,  the  HM3  system  produces  data  with  lower  spatial  density  from 
magnetometers  that  are  deployed  at  a  higher  altitude.  Typical  sensor  height  above  ground 
is  2  m,  sensor  spacing  is  6  m,  and  typical  along-track  data  density  is  0.5  m  per  point. 
Interleaved  tracks  were  flown  so  that  data  lanes  were  separated  by  a  nominal  3  m.  With 
lower  sample  density  and  helicopter  speeds  (~45  kts),  the  HM3  system  provides  much 
faster  site  coverage  than  is  achievable  with  a  ground-based  system.  The  HM3  system 
visited  five  live  areas  on  BBR  plus  a  seeded  calibration  site.  Two  of  the  live  areas  were 
the  BBR1  and  BBR2  sites  visited  2  years  previously  by  the  MTADS  system.  This  study 
will  concentrate  on  the  BBR1  and  BBR2  surveys  for  platform  comparisons  and  on  the 
calibration  area  to  explore  sensitivity  on  a  broader  range  of  targets.  Details  of  the  HM3 
sensor  and  the  ORNL  site  activities  are  provided  in  Reference  2.  The  HM3  survey 
produced  dig  lists  of  520  and  296  targets  nominated  for  digging  on  BBR1  and  BBR2, 
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respectively.  Intrusive  sampling  was  performed  on  17  targets  on  BBR1  and  24  targets  on 
BBR2. 

This  work  is  an  independent  evaluation  of  the  performance  of  these  two  advanced 
sensor  systems,  both  individually  and  in  comparison  to  one  another.  First,  we  look  at  data 
from  the  two  individual  systems.  In  the  reports  generated  by  the  testers,  there  are  some 
differences  in  the  way  that  performance  is  scored  and  described.  This  is  partly  attributed 
to  semantics  and  partly  attributed  to  the  different  missions  that  were  originally  envisioned 
for  the  two  systems:  the  MTADS  would  be  used  primarily  to  do  detailed  surveys  to 
generate  dig  lists  and  the  HM3  primarily  to  do  wide-area  searches  to  locate  concentra¬ 
tions  of  ordnance,  such  as  in  impact  areas,  but  not  necessarily  to  generate  dig  lists. 
Success  of  these  two  missions  would  be  measured  by  different  performance  metrics. 
Chiefly,  the  scrap  that  is  properly  considered  clutter  (false  alarms)  in  a  dig  list  may 
provide  as  good  a  clue  to  the  presence  of  an  impact  area  as  intact  ordnance;  however, 
HM3  results  were  quite  good  and  led  us  to  evaluate  its  ability  to  generate  dig  lists.  So,  in 
addition,  we  look  at  the  differences  in  performance  for  the  detection  of  individual  targets 
and  the  associated  false  alarms  and  examine  the  data  for  differences  caused  by  the  lower 
sampling  density  and  higher  altitude  of  the  HM3  sensor. 

B.  ANALYSIS  OF  NRL  MTADS  DATA 

MTADS  surveyed  the  BBR  site  using  magnetometer  and  EMI  systems.  For 
comparison  to  the  HM3  data,  this  study  focuses  on  the  magnetometer  data.  The  MTADS 
data  processing  fits  the  spatial  magnetic  signature  to  estimate  the  depth  and  size  of  the 
object.  First,  we  examined  the  dig  list  targets  that  were  excavated  to  establish  ground 
truth  following  the  demonstration.  On  BBR1,  146  items  were  excavated,  all  north  of  the 
fence  line,  and  on  BBR2,  225  items  were  excavated,  yielding  a  substantial  body  of 
ground  truth  data.  We  examined  the  target  features  extracted  by  NRL  testers  and  the  raw 
signal  data  for  comparison  with  the  ORNL  results. 

The  BBR1  data  represents  an  easier  case  for  discrimination,  with  both  the  target 
and  the  clutter  populations  fairly  well  defined.  The  former  consisted  entirely  of  M-38 
practice  bombs  and  the  latter  mostly  scrap  from  these  items.  As  such,  the  targets  and 
clutter  displayed  very  different  characteristics.  The  ground-truthed  items  were  divided 
into  clutter  and  ordnance.  The  derived  features  (which  include  magnetic  moment,  fit 
quality,  depth,  and  size)  from  the  NRL  data  analysis  were  examined  to  determine  if  any 
provided  reliable  discrimination.  The  results  are  shown  in  Figure  1.  Size  and  the  related 
moment  provide  fairly  reliable  discriminants,  but  for  the  other  parameters,  targets  and 
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clutter  populations  are  not  as  well  separated.  From  here  forward,  size  is  used  for  discrimi¬ 
nation  analyses. 

In  Table  1,  a  size  threshold  of  0.125  is  applied  to  the  known  target  and  clutter 
items.  This  threshold  gives  a  probability  of  correct  identification  of  ordnance,  given  a 
detection,  of  0.97  and  a  probability  of  false  positive  of  0.20.  Previous  MTADS  tests  show 
good  probability  of  detection  (near  100  percent)  on  bombs  (Refs.  3  and  4).  We  conclude 
that  the  likelihood  of  missing  a  bomb  on  this  site  using  this  threshold  is  low,  that  setting  a 
threshold  of  0.125  m  on  size  will  retain  most  bombs  in  the  dig  lists,  but  that  any  list  of 
ordnance-like  objects  using  the  best  descriminant  we  have  been  able  to  identify  will 
inevitably  contain  some  (20  percent)  clutter.  This  site  likely  represents  the  best  possible 
opportunity  for  discrimination  because  of  the  homogeneity  of  ordnance  types  and  the 
limited  clutter  sources. 


Table  1.  Size  Threshold  Applied  to  MTADS  Ground  Truth  on  BBR1N 


MTADS  size  threshold 
says 

Total 

Is  Bomb 

Is  Not  Bomb 

Bomb  (>0.125) 

84 

71 

13 

Not  bomb  (<0.125) 

55 

2 

53 

Total 

73 

66 

Many  of  the  targets  selected  for  the  dig  lists  in  the  MTADS  data  analysis  were  not 
excavated.  The  targets  that  were  not  dug  on  the  north  side  of  the  fence  line,  as  well  as  all 
the  targets  picked  on  south  side  of  fence  line  where  no  digging  was  possible,  were 
considered  using  the  size  discriminant.  The  distribution  plot  for  the  dig  list  targets  that 
were  not  dug  on  the  north  side  of  the  fence,  shown  in  the  upper  portion  of  Figure  2,  is 
very  similar  to  that  for  the  ground-truthed  targets  north  of  fence.  The  same  bimodal 
distribution,  which  is  noted  in  Reference  1,  suggests  a  similar  separation  into  M-38 
targets  and  clutter.  On  the  south  side  of  the  fence,  the  relative  abundances  of  apparent 
targets  and  apparent  clutter  differ  from  what  was  observed  in  the  dug  targets  on  the  north 
side.  This  is  not  surprising  since  the  area  south  of  the  fence  has  been  cultivated  for  years. 
The  cultivation  likely  either  brought  small  clutter  items  to  the  surface,  where  they  were 
cleared  by  the  farmer,  or  buried  them  beyond  detection  range.  NRL  anecdotally  reported 
an  accumulation  of  scrap  along  the  fence  line  that  is  seen  in  magnetometer  images 
produced  by  both  systems,  which  was  presumably  amassed  as  scrap  objects  were  tossed 
aside  during  farming  activities. 
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Figure  1.  BBR1  Ordnance  and  Clutter  Population  Distributions  for  MTADS-Derived  Parameters 
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Figure  2.  MTADS  Size  Distributions  for  Targets  Not  Dug  on  BBR1.  The  upper  plot  is 
for  the  north  side  of  the  fence  line  and  the  lower  plot  is  for  the  south  side. 

Based  on  the  size  analysis,  all  the  MTADS  dig  list  targets  on  the  BBR1  site  were 
divided  into  bomb-like  and  clutter-like,  with  the  threshold  set  at  0.125  m  for  subsequent 
analyses. 

The  data  for  BBR2  is  more  difficult  to  interpret.  This  site  contains  a  greater 
variety  of  ordnance:  the  targets  dug  by  MTADS  include  M-38,  small-caliber  artillery 
rounds  (SCARs),  and  2.75-in.  rockets,  with  one  20-mm  and  one  hole  containing  two 
targets.  We  attempted  to  do  the  same  exercise  for  discriminants  and  were  not  surprised 
that  it  was  less  successful  on  BBR2.  Figure  3  shows  the  dug  targets  separated  into 
ordnance  and  nonordnance  using  the  size  discriminant.  The  profile  of  the  clutter  on 
BBR2  is  not  very  different  from  that  found  on  BBR1,  but  the  ordnance  does  not  neatly 
separate  out.  If  the  ordnance  is  separated  by  type,  the  size  distributions  are  as  shown  in 
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Figure  4.  The  SCARs  cover  a  wide  range  of  sizes  and  the  distribution  of  2.75  rocket 
warheads  significantly  overlaps  the  clutter.  The  M-38  bomb  size  distribution  is  not  very 
different  from  what  was  observed  for  these  targets  in  BBR1,  although  the  centroid  is 
concentrated  at  slightly  larger  sizes.  With  far  fewer  M-38s  dug  on  BBR2,  it  is  not  clear 
that  this  represents  any  statistically  meaningful  shift  in  the  population,  which  would  be 
unlikely  in  any  event. 
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Figure  3.  BBR2  MTADS  Size  Distributions  of  Target  and  Clutter 
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Figure  4.  BBR2  MTADS  Size  Distributions  Broken  Out  by  Ordnance  Type 
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C.  ANALYSIS  OF  ORNL  HM3  DATA 


We  performed  two  types  of  analyses  using  the  ORNL  data.  First,  using  the  ORNL 
picks,  we  looked  at  characteristics  in  analytic  signal  in  an  attempt  to  determine  if  any 
discrimination  could  be  performed  based  on  the  analysis  as  presented.  Second,  in  the  next 
section,  to  compare  to  MTADS  data  and  to  evaluate  sensitivity  of  the  system  based  on  the 
calibration  area,  we  looked  at  the  raw  magnetometry  data. 

The  rescoring  of  the  HM3  data  was  performed  to  select  dig  lists  from  the  data. 
When  originally  scored  by  ORNL,  all  magnetic  returns  that  were  reacquired  by  ground- 
based  sensors  were  regarded  as  successful  hits,  regardless  of  whether  they  represented 
intact  ordnance,  ordnance-related  scrap,  or  even  hot  soils.  For  the  most  part,  this  is 
reasonable  (except  perhaps  for  the  inclusion  of  hot  soils):  since  the  goal  of  the  work  was 
to  characterize  impact  areas,  any  ordnance-related  scrap  could  be  considered  a  legitimate 
target.  To  generate  dig  lists,  however,  we  are  interested  in  finding  intact  ordnance,  versus 
fins  and  rocket  motors,  which  do  not  present  a  hazard.  Finally,  we  also  need  to  select  a 
common  terminology  and  scoring  system  to  do  the  comparisons  to  MTADS  performance. 
We  opt  for  the  scoring  system  shown  in  Table  2.  Here,  we  eliminate  the  use  of  the  term 
Pd  in  Reference  2,  since  the  total  number  of  ordnance  items  encountered  is  not  known  (Pd 
therefore  cannot  be  determined),  and  replace  it  with  percent  ordnance  (the  fraction  of  digs 
that  resulted  in  ordnance).  The  primary  result  of  the  rescoring  is  that  the  percentage  of 
targets  corresponding  to  ordnance  (referred  to  as  Pd  in  Ref.  2)  is  lower,  and  the 
percentage  of  false  positives  is  higher. 


Table  2.  ORNL  HM3  Results  as  Rescored  by  IDA 


Area 

Anoms 

Digs 

M38 

2.25  in. 

Frag  or 
Scrap 

Hot 

Soils 

No 

Contact 

%  Ord 

%  FP 

1 

49 

17 

13 

0 

1 

1 

2 

76 

24 

2 

33 

24 

16 

4 

4 

0 

0 

83 

17 

We  begin  with  a  brief  summary  of  the  HM3  data.  The  system  records  full- field 
magnetometer  data,  which  is  processed  using  the  Geosoft  tool  to  obtain  what  is  referred 
to  as  an  analytic  signal,  essentially  a  horizontal  gradient.1  All  decisions  regarding  target 
declarations  were  made  by  the  ORNL  team  on  the  basis  of  the  analytic  signal.  Initial 
target  selection  was  done  by  amplitude  thresholding  on  the  analytic  signal  and  selecting 


1  The  analytic  signal  is  computed  as  as  =  \dT/dxf  +  (dT/dyf  +  (dT/dzf  f*  where  T  is  the  total  field.  The  x  and 
y  derivatives  are  computed  spatially  from  the  grid,  and  the  z  derivative  is  computed  using  a  Fourier 
transform. 
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everything  over  1.0  nT/m  as  a  potential  target.  From  this  list,  targets  were  selected  for  a 
dig  list,  which  was  then  further  downselected  for  actual  digs.  The  number  of  dug  targets 
was  extremely  limited  and  spanned  the  range  of  signal  amplitudes  (i.e.,  the  dug  targets  do 
not  represent  all  the  strongest  signals). 

If  the  goal  is  to  generate  dig  lists,  it  will  be  important  to  identify  a  discriminant  to 
distinguish  ordnance  from  clutter.  In  the  case  of  the  HM3  data,  the  only  parameter 
calculated  in  the  original  data  analysis  was  analytic  signal.  Unfortunately,  few  targets 
were  dug  following  the  HM3  survey.  On  BBR1,  only  17  excavations  were  conducted, 
and  all  were  in  the  part  of  the  site  north  of  fence  line.  Figure  5  shows  distributions  of 
analytic  signal  for  ordnance  and  nonordnance  ground  truth.  There  is  no  potential  to  draw 
a  line  separating  the  two  populations,  which  significantly  overlap.  This  finding  is  not 
surprising,  since  it  involved  no  more  than  creating  an  amplitude  threshold:  depending  on 
target  size  and  depth  and  sensor  geometry,  one  expects  the  amplitudes  to  span  the  space. 


Analytic  Signal  (nT/m) 


Figure  5.  BBR1  HM3  Analytic  Signal  Distribution 

The  results  for  the  analytic  signal  on  BBR2  are  similar.  As  shown  in  Figure  6,  the 
ordnance  and  nonordnance  (clutter)  show  almost  total  overlap  in  this  parameter.  The  dug 
targets  on  BBR2  included  only  M-38  practice  bombs  and  2.25-in.  rockets. 
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Figure  6.  BBR2  HM3  Analytic  Signal  Distribution 

D.  CROSS  COMPARISONS— WHO  FOUND  WHAT? 

Figure  7  shows  the  targets  selected  by  MTADS  and  the  HM3  on  BBR1.  This  plot 
includes  all  targets  picked  by  each  sensor,  regardless  of  whether  they  were  dug  or 
whether  the  target  attributes  indicated  that  they  were  clutter-like  or  bomb-like.  The  linear 
feature  that  runs  approximately  along  the  Y  =  0  line  is  the  fence  separating  the  north  and 
south  sides.  The  MTADS  target  picks  are  separated  into  those  dug  and  not  dug.  The  plot 
shows  some  commonality  of  targets.  This  is  an  important  initial  result:  the  two  data  sets 
were  not  submitted  in  common  coordinate  systems,  so  some  coordinate  conversions  were 
required  to  enable  comparisons.  Both  data  sets  were  originally  taken  in  using  GPS  in 
WGS84  latitude  and  longitude;  however,  ORNL  used  the  NAD27  datum  for  transforma¬ 
tion  to  Universal  Transverse  Mercator  (UTM)  coordinates,  and  NRL  used  NAD83.  Initial 
attempts  at  direct  conversion  between  the  two  data  sets  were  not  successful,  resulting  in 
significant  offsets.  The  likely  cause  of  the  difficulty  was  an  error  in  the  NAD27  data  used 
by  ORNL,  which  was  noted  in  the  original  report  by  the  performers,  but  the  root  cause  of 
it  was  never  determined.  We  requested  that  ORNL  provide  locations  of  target  picks  in  the 
original  lat/long  WGS84  and  successfully  used  a  commercial  software  package  by  Blue 
Marble  Geographies  (Ref.  5)  to  do  the  conversion  to  NAD83.  Then,  ORNL  NAD83 
coordinates  were  transformed  into  MTADS  local  coordinates  for  convenience,  and  all 
subsequent  analyses  were  performed  on  these  data  sets.  After  the  initial  difficulties,  the 
matching  of  targets  between  the  two  data  sets  gave  us  confidence  that  the  conversions 
were  done  correctly. 
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All  performance  comparisons  in  this  document  must  be  considered  in  the  context 
of  the  philosophies  of  target  selection  employed  by  the  two  performers,  which  may  skew 
the  results  considerably.  The  MTADS  target  list  on  BBR1  includes  the  initial  targets 
selected  for  the  training  set,  as  well  as  targets  from  the  entire  site  selected  based  on  the 
training  set  results.  Target  picks  on  the  entire  site  were  not  weeded  to  select  only  large 
bombs,  but  to  mirror  the  profile  of  the  training  set.  On  BBR2,  the  MTADS  list  con¬ 
centrated  on  sampling  smaller  items.  The  HM3  targets  were  selected  using  a  simple 
amplitude  threshold  in  the  analytic  signal  data.  Obvious  nonordnance  sources  of 
anomalies,  such  as  fences  and  posts,  were  manually  removed  from  the  dig  lists. 

Figure  8  shows  the  MTADS  and  HM3  picks  on  the  north  side  of  the  fence  in 
BBR1.  Again,  the  MTADS  target  picks  are  divided  into  those  that  were  dug  and  not  dug. 
Recall  that  the  MTADS  targets  were  dug  prior  to  the  HM3  survey,  so  none  of  the 
MTADS  dug  targets  is  expected  to  be  observed  in  the  HM3  data.  The  MTADS  targets 
that  were  not  dug  were  separated  into  bomb-like  and  clutter-like  based  on  the  size 
threshold  of  0.125  m  selected  above.  These  locations  were  then  examined  for  coincident 
target  declarations  in  the  HM3  data.  Table  3  shows  the  results. 

Of  the  172  bomb-like  targets  MTADS  detected  but  did  not  excavate,  95  were 
detected  by  the  HM3  (55  percent).  Of  the  173  clutter-like  targets  MTADS  detected  but 
did  not  excavate,  only  13  (8  percent)  were  detected  by  the  HM3. 

Figure  9  shows  the  MTADS  and  HM3  target  picks  for  BBR1  south  of  the  fence 
line,  none  of  which  were  excavated.  Again,  the  MTADS  picks  are  divided  using  the  size 
threshold  into  bomb-like  (>0.125)  and  clutter-like  (<0.125)  and  analyzed  as  for  the  north 
side.  Table  4  shows  the  results.  The  outcome  is  similar,  with  HM3  detecting  45  percent  of 
the  bomb-like  MTADS  targets  and  8  percent  of  the  clutter-like  MTADS  targets. 
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Figure  7.  BBR1  MTADS  and  ORNL  Dig  List  Picks.  All  targets  are  plotted  in  the  MTADS  local  X  and  Y  coordinates,  where 
X  runs  east-west,  Y  runs  north-south,  and  (0,0)  is  located  approximately  at  the  center  of  the  bombing  target  bull’s-eye. 
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Figure  8.  BBR1  MTADS  and  HM3  Dig  List  Picks  North  of  the  Fence 


Table  3.  BBR1  North  MTADS  Target  Picks  Not  Excavated  vs.  HM3  Target  Picks1 


MTADS 

HM3  Target 

No  HM3  Target 

Bomb-like  (size  >0.125) 

172 

95 

77 

Clutter-like  (size  <0.125) 

173 

13 

160 

1  Based  on  a  threshold  of  1  nT/m  in  the  analytic  signal. 
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Figure  9.  BBR1  MTADS  and  HM3  Target  Picks  South  of  the  Fence 


Table  4.  BBR1  South  MTADS  Target  Picks  vs.  HM3  Target  Picks 


MTADS 

HM3  Target 

No  HM3  Target 

Bomb-like  (size  >0.125) 

170 

77 

93 

Clutter-like  (size  <0.125) 

48 

4 

44 

We  also  compared  the  HM3  dig  results  with  the  MTADS  targets  not  dug  for 
coincident  detections.  Figure  10  shows  the  HM3  dug  targets  labeled  to  show  the  ground 
truth  determined  by  digging  at  BBR1.  Overlaid  on  the  HM3  ground  truth  and  marked  as 
bomb-like  and  clutter-like  are  the  MTADS  targets  not  dug.  Note  that  few  targets  were 
dug  based  on  the  HM3  picks  and  further,  no  effort  was  made  to  pick  targets  believed  to 
be  bombs.  The  results  are  summarized  in  Table  5.  MTADS  visited  a  smaller  area  than  the 
HM3.  Ten  of  the  eleven  practice  bombs  found  in  the  FIM3  ground  truthing  in  the 
common  area  were  detected  and  selected  for  the  MTADS  target  list.  The  bomb  at 
approximately  X  =  -150  m,  Y  =  100  m,  which  does  not  appear  in  the  MTADS  target  list, 
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is  in  an  area  where  the  MTADS  anomaly  map  shows  a  small  patch  of  ground  where  no 
data  was  taken  because  of  an  obstacle  to  driving.  With  MTADS  capable  of  a  higher 
sample  density  and  making  measurements  at  significantly  closer  standoff  distance,  it 
would  be  surprising  for  MTADS  to  miss  anything  found  by  HM3.  The  MTADS  did  not 
select  any  targets  at  the  locations  where  the  HM3  ground  truth  resulted  in  no  contacts  or 
fins.  At  the  location  of  the  single  “hot  soil”  reported  by  the  HM3,  the  MTADS  reported  a 
target  which  falls  into  the  “clutter-like”  category  base  on  size  discrimination. 
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Figure  10.  HM3  Ground  Truth  and  MTADS  Dig  List  Picks  Not  Dug 

Table  5.  BBR1  HM3  Ground  Truth  from  Excavated  Targets  vs. 
MTADS  Targets  Not  Excavated 


HM3 

MTADS 

Bomb-like 

Clutter-like 

No  Detect 

No  Visit 

M38 

13 

10 

0 

1 

2 

No  Contact 

2 

0 

0 

2 

0 

Hot  Soil 

1 

0 

1 

0 

0 

Fins 

1 

0 

0 

1 

0 

The  same  analysis  was  done  on  BBR2.  We  compared  the  HM3  dug  targets  with 
the  MTADS  targets  picks  that  were  not  dug.  Figure  11  shows  a  plot  of  all  the  dig  list 
picks  with  HM3  ground  truth  indicated.  The  results  are  summarized  in  Table  6.  Because 
of  the  lack  of  success  in  identifying  a  reliable  discriminant  in  the  MTADS  data  for  BBR2, 
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no  attempt  was  made  to  differentiate  MTADS  picks  into  ordnance-like  and  nonordnance¬ 
like.  Of  21  total  ordnance  items  found  in  the  HM3  ground  truthing,  MTADS  had  the 
opportunity  to  detect  10  (i.e.,  the  locations  of  the  other  1 1  items  were  not  visited  in  the 
MTADS  survey).  Of  these  10  opportunities,  MTADS  reported  target  picks  at  9.  The 
missed  item  was  a  rocket. 
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Figure  11.  BBR2  HM3  Ground  Truth  and  MTADS  Target  Picks  Not  Dug 
Table  6.  BBR2  HM3  Ground  Truth  vs.  MTADS  Targets  Not  Excavated 


HM3 

MTADS 

Detect 

No  Detect 

No  Visit 

M38 

14 

6 

0 

8 

Rockets 

5 

1 

1 

3 

Mixed 

2 

2 

0 

0 

Clutter 

3 

0 

1 

2 

E.  TARGET  COMPARISONS— CAN  WE  SAY  WHY? 

Further  coordinate  conversion  problems  were  encountered  in  the  raw  data 
analysis.  We  were  supplied  with  the  NAD27  UTM  coordinates  for  the  raw  data,  and 
attempts  to  convert  directly  from  NAD27  to  NAD83  resulted  in  an  offset.  We  used 
several  points  widely  spaced  in  the  image  to  estimate  the  true  offset  by  comparing  their 
locations  to  the  correct  locations  of  targets  in  the  dig  list  derived  from  direct  conversion 
of  the  WGS84  GPS  data  described  above.  Additional  corrections  were  applied  to  the  raw 
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data  locations.  These  corrected  UTM  NAD27  locations  were  then  transformed  into 
NAD83,  which  were  then  transformed  into  MTADS  local  coordinates  for  analysis. 

1.  Site  Coverage 

Given  the  sensor  layout  and  modes  of  operation,  the  data  densities  for  MTADS 
and  the  HM3  are  very  different.  We  attempt  to  quantify  the  effect  of  this  difference  and 
examine  its  impact  on  data.  Consider  BBR1.  The  site  is  gridded  into  squares,  beginning 
with  1  m  on  a  side  and  growing  to  5  m  on  a  side.  For  each  grid  size,  the  fraction  of  the 
grid  squares  visited  by  each  sensor  was  counted  (see  Table  7).  Figure  12  shows  HM3 
coverage  rates  for  various  grid  sizes. 


Table  7.  Coverage  Rates  for  Various  Grid  Sizes 


Grid  Size  (m) 

MTADS 

HM3 

1 

99.9 

33.1 

2 

100.0 

58.8 

3 

100.0 

76.4 

4 

100.0 

88.1 

5 

100.0 

95.0 

2.  Individual  Signature  Comparisons 

A  few  targets  were  selected  for  direct  comparison  of  signatures  in  the  sensor 
data.  These  included  two  M-38  bombs,  one  at  a  depth  of  3.5  ft  (1.1  m)  and  the  other  at 
1.5  ft  (0.5  m),  and  one  false  alarm  that  corresponded  to  hot  soil.  Figures  12-14  show 
chips  of  the  HM3  total  field  data,  the  HM3  analytic  signals,  and  the  MTADS  magne¬ 
tometer  data  for  the  selected  targets.  First,  we  considered  the  maxima  in  the  total  field 
signals,  which  are  shown  in  Table  8.  Since  none  of  these  signals  has  magnitude  near  the 
noise  floor  of  the  instruments,  the  differences  in  maximum  signals  should  be  dominated 
by  the  proximity  of  the  sensors  to  the  targets.  From  the  vertical  component  of  the  distance 
alone  (i.e.,  depth  of  target  plus  height  of  sensor),  there  is  a  factor  about  10  to  about  30 
difference  in  expected  signal  strength  between  the  HM3  and  the  MTADS  total  field 
measurements  for  these  targets.  The  lower  spatial  density  of  the  HM3  data  on  average 
will  easily  increase  this  significantly.  Other  effects  such  as  the  relative  orientation  of  the 
sensors  and  targets  are  not  considered,  but  are  likely  to  be  significant.  Generally,  a  factor 
of  about  100  or  more  difference  is  seen  in  the  maxima. 
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Northing  (m)  _  Northing  (m)  _  Northing  (m) 


MTADS  Total  Magnetic  Signal 


ORNL  Total  Magnetic  Signal 
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igure  12.  HM3  Total  Magnetic  Signal,  HM3  Analytic  Signal,  and  MTADS  Magnetic  Signal 
for  BBR1  HM3  Target  1039,  an  M-38  Practice  Bomb  Buried  at  1.1  m 
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igure  13.  HM3  Total  Magnetic  Signal,  HM3  Analytic  Signal,  and  MTADS  Magnetic  Signal 
for  BBR1  HM3  Target  1064,  an  M-38  Practice  Bomb  Buried  at  0.5  m 
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Figure  14.  HM3  Total  Magnetic  Signal,  HM3  Analytic  Signal,  and  MTADS  Magnetic  Signal 
for  BBR1  HM3  Target  1380,  Which  Resulted  from  Magnetic  Soil  at  1.3  m 
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Table  8.  Summary  of  Sensor  Data  for  Selected  Targets  on  BBR1 


Target  # 
(ORNL) 

Type 

Depth  (m) 

Sensor 

Distance 

Ratio1 

MTADS 

Mag 

(nT) 

HM3 

Mag 

(nT) 

HM3 

Analytic  Signal 
(nT/m) 

1039 

M38 

1.1 

13 

3967 

22 

15 

1064 

M38 

0.5 

33 

1694 

18 

12 

1340 

Hot  soil 

1.3 

11 

487 

5 

2.7 

1  Sensor  distance  ratio  is  calculated  using  the  vertical  dimension  only.  The  height  of  the  sensor 
above  ground  is  added  to  the  target  depth,  and  the  ratio  of  the  HM3  distance  to  the  MTADS 
distance  is  cubed. 


F.  LOCATION  ACCURACY 

As  reported  in  Reference  1,  the  MTADS  location  accuracy,  gives  an  average  miss 
distance  of  12  cm,  with  90  percent  of  the  targets  detected  within  22  cm.  No  similar 
analysis  is  provided  in  Reference  2  for  the  HM3  system;  however,  because  of  the 
difference  in  data  densities,  we  expect  considerably  larger  location  errors  compared  with 
MTADS.  Using  the  39  targets  excavated  on  BBR1  and  BBR2,  we  calculated  that  the 
average  positional  error  was  0.5  m  and  that  90  percent  of  the  targets  were  found  within 
1  m. 


To  estimate  the  relative  accuracy  of  the  HM3  and  MTADS,  we  examined  an  area 
of  BBR1  with  a  relatively  low  density  of  picked  targets  and  set  a  10-m  distance  as  a 
cutoff  in  matching  the  HM3  picks  to  the  MTADS  picks.  Within  the  resulting  matches,  the 
mean  distance  from  MTADS  to  HM3  location  is  3.0  m,  the  maximum  radial  offset  occurs 
at  7  m,  and  only  7  percent  of  the  hits  cross  correlate  at  1  m  separation.  This  is  in  stark 
contrast  to  the  accuracy  observed  in  the  HM3  ground  truthing.  We  have  been  unable  to 
unambiguously  identify  the  cause  of  this  discrepancy,  although  it  likely  lies  in  the 
coordinate  conversion  difficulties  described  above.  Residual  errors  in  the  coordinate 
conversion  process  could  result  from  our  calculations,  although  this  seems  unlikely  for 
the  target  pick  locations  used  in  the  location  analyses  because  their  X,Y  coordinates  were 
calculated  directly  from  the  WGS84  lats  and  longs,  which  we  have  no  reason  to  believe 
contained  errors.  Another  possibility  is  that  any  systematic  errors  in  the  HM3  location 
data  were  obscured  by  the  empirical  location  corrections  applied  at  the  time  of  the 
excavations. 
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G.  PRODUCTION  RATES  AND  COSTS 


Accurate  estimates  of  production  rates  and  costs  will  be  difficult  to  obtain  for 
both  systems  from  this  demonstration  because  there  is  no  accounting  breakdown  of  time 
spent  on  various  tasks.  This  presents  several  problems:  the  MTADS  demonstration 
included  an  EMI  survey  and  the  time  to  analyze  these  data  is  not  broken  out;  the  level  of 
analysis  done  by  the  two  performers  was  quite  different,  both  in  terms  of  the  type  of  anal¬ 
ysis  and  the  previous  experience;  and  the  number  of  targets  excavated  differed  greatly. 

We  attempted  to  segregate  the  impact  of  a  towed  platform  versus  an  airborne 
platform  on  survey  rate  from  the  other  unknown  parameters.  To  do  this,  we  looked  at  the 
specifications  of  both  systems  as  reported  in  this  demonstration  and  previous  MTADS 
demonstrations  to  estimate  survey  time  on  a  relatively  large  site  of  1  ktrf.  For  the  HM3, 
we  use  the  quoted  helicopter  speed  of  20  m/sec,  the  sensor  spacing  of  6  m  on  the 
platform,  and  the  interleaved  survey  pattern  to  give  3-m  nominal  data  line  spacings.  With 
these  parameters,  1.8  hours  would  be  required  to  survey  1  km  .  This  estimate  does  not 
include  additional  time  required  for  turns.  For  MTADS,  we  use  the  2-m  array  width  (with 
tracks  correspondingly  separated  by  2  m)  and  a  travel  speed  of  2.5  m/s  to  estimate  that 
the  1  km  site  would  require  55  hours  of  survey  time,  also  not  including  turn-around  time. 

For  this  demonstration,  the  MTADS  total  cost  was  $377,296.  If  the  excavation 
costs  of  $169,096  and  the  reporting  costs  of  $24,000  are  removed,  the  MTADS  costs  for 
the  deployment,  survey,  and  analysis  parts  of  this  demonstration  were  $184,200.  (Note 
that  this  does  not  separate  out  the  costs  of  the  EMI  work.)  The  MTADS  surveyed  a  total 
of  more  than  150  acres  for  a  cost  of  $1,222  per  acre.  For  the  HM3,  the  total  costs  of  this 
demonstration  were  $220,000  to  survey  287  acres,  for  a  cost  of  $766  per  acre. 

Both  performers  in  their  respective  demonstration  reports  made  estimates  of  the 
time  and  cost  required  to  survey  large  sites.  These  are  summarized  in  Table  9.  We  used 
the  MTADS  projections  for  a  1,000-acre  site.  We  did  not  do  any  analysis  to  verify  the 
projections  of  either  performer.  As  would  be  expected,  the  projected  costs  per  acre  for 
both  performers  are  lower  for  production  work  than  those  incurred  for  this  demonstration. 


Table  9.  Summary  of  Performer  Projections  of  Cost  and  Production  Rate 
for  Large  Sites  (Refs.  1  and  2) 


Cost  ($/acre) 

Production  Rate  (acres/day) 

MTADS  (1,000-acre  site) 

570 

10 

HM3 

200 

200 

20 


The  specific  projection  numbers  probably  do  not  have  much  meaning  as  they  will 
depend  on  the  size  of  the  site,  the  type  and  density  of  ordnance  contamination,  the 
deployment  costs,  and  many  other  parameters  in  any  real  clean-up  operation.  There  is  no 
doubt,  however,  that  the  helicopter  platform  will  be  faster  by  a  factor  of  10  or  more,  and 
it  is  also  likely  to  be  less  expensive. 

H.  CONCLUSIONS 

First,  it  is  important  to  note  that  the  most  appropriate  use  for  these  technologies 
may  be  in  different  missions.  The  HM3  mounted  on  a  helicopter  was  originally  conceived 
as  a  footprint-reduction  tool,  where  the  main  requirement  was  to  detect  enough  ordnance 
or  ordnance-related  debris  to  identify  areas  that  warrant  more  thorough  examination.  On 
the  other  hand,  the  MTADS’  role  is  to  produce  dig  lists  of  individual  targets.  As  such,  the 
ability  to  do  discrimination  is  not  equally  important  to  the  two  systems.  In  fact,  for  the 
footprint-reduction  mission,  finding  ordnance-related  clutter  may  provide  as  important  an 
indication  of  an  impact  area  as  finding  intact  ordnance. 

The  results  of  the  HM3,  however,  are  very  good,  and  it  is  tempting  to  ask  how  it 
would  do  in  producing  dig  lists.  This  report  looks  at  what  is  possible  using  the  current 
data: 

•  On  a  homogeneous  site,  the  Pd  achieved  by  the  HM3  on  individual  targets  is 
about  50  percent  of  that  achieved  by  the  MTADS. 

•  Only  about  8  percent  of  the  apparent  clutter  that  appears  in  the  MTADS  dig 
lists  is  reported  by  the  HM3. 

•  Of  22  ordnance  items  detected  and  confirmed  in  the  ground  truth  by  HM3, 
20  were  detected  by  MTADS.  The  cause  of  one  missed  detection  is  likely 
inaccessibility  of  the  area  by  the  vehicle. 

•  The  helicopter-mounted  HM3  provides  much  faster  production.  We  roughly 
estimate  that  the  HM3  survey  rate  would  be  about  10  times  faster  than  the 
MTADS  rate  for  a  large  site. 

•  Cost  estimates  prepared  by  the  performers  indicate  that  the  per  acre  cost  of 
the  MTADS  is  about  2-3  times  higher  that  those  of  the  HM3.  These  figures 
are  very  rough  estimates  and  may  not  accurately  reflect  cost  differences  seen 
in  operational  surveys. 

In  the  process  of  this  analysis,  we  have  encountered  several  questions  that  could 
not  be  addressed  adequately  with  that  data  as  it  was  collected.  This  is  not  surprising, 
since  the  purpose  of  neither  data  collection  was  to  provide  comparison  data. 
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Nevertheless,  we  offer  some  suggestions  for  future  similar  efforts  that  would  allow  for 
more  complete  analysis. 

•  The  ground  truth  from  the  HM3  excavated  targets  was  inadequate,  both  for 
evaluation  of  this  system  and  for  comparison  with  the  MTADS.  Although 
extensive  ground  truth  sampling  is  expensive,  it  is  critical  to  obtaining 
meaningful  data  for  live  demonstrations.  Sampling  of  at  least  100  targets  in 
future  demonstrations  is  recommended. 

•  The  elapsed  time  of  2  years  between  the  MTADS  and  HM3  demonstrations 
likely  had  some  impact  on  the  system  comparisons.  This  site  was  in  use 
during  the  intervening  time,  when  items  producing  magnetic  anomalies  may 
have  been  added,  removed,  or  displaced  by  farming  activities. 

•  Conducting  excavations  between  the  two  demonstrations  caused  complica¬ 
tions.  Obviously,  it  was  a  great  expenditure  of  resources  that  provided 
ground  truth  for  only  one,  instead  of  both,  demonstrations.  It  also  made  data 
analysis  for  comparisons  more  time  consuming,  requiring  the  tedious 
separation  of  MTADS  targets  to  isolate  those  left  in  the  ground  and  available 
to  be  detected  by  the  HM3 . 

•  Finally,  the  selection  of  targets  for  ground  truthing  should  be  performed  with 
a  well  articulated  purpose  in  mind.  It  is  difficult  to  make  meaningful 
comparisons  if  the  goal  of  one  system  is  to  maximize  ordnance  picks  and  that 
of  the  other  is  to  span  the  range  of  signals  for  analysis  purposes. 


22 


GLOSSARY 


BBR  Badlands  Bombing  Range 

DGPS  differential  Global  Positioning  System 

EMI  electromagnetic  induction 

ESTCP  Environmental  Security  Technology  Certification  Program 

HM3  high-sense  helicopter-mounted  magnetic  mapping 

MTADS  Multisensor  Towed-Array  Detection  System 

NRL  Naval  Research  Laboratory 

ORNL  Oak  Ridge  National  Laboratory 

SCAR  small-caliber  artillery  round 

UTM  Universal  Transverse  Mercator  map  coordination  system 
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